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' Abstract 

(~| ' We describe an algebra for composing automata in which the actions 

, have probabilities. We illustrate by showing how to calculate the proba- 

bility of reaching deadlock in k steps in a model of the classical Dining 
' Philosopher problem, and show, using the Perron-Frobenius Theorem, 

that this probability tends to 1 as fc tends to infinity. 

O ; 1 Introduction 

h- j 

C/3 I The idea of this paper is to introduce into the algebra of automata introduced 

, in [3] probabiUties on the actions. This permits a compositional description of 

probabilistic processes, in particular of Markov chains, and for this reason we 
call the automata we introduce Markov automata. 
^ I We define a Markov automaton with a given set A of "signals on the left 

interface" , and set B of "signals on the right interface" to consist of a Markov 
matrix Q (whose rows and columns are the states) which is the sum 



(N 



Q = Qai,bi+Qa2,6i+--- + Qa„ 



■ of non- negative matrices Qatfij whose elements are the probabilities of transi- 
I tions between states for which the signals Oi and bj occur on the interfaces. 

In addition, each alphabet is required to contain a special symbol e (the null 
signal) and the matrix Qe^^eB is required to have row sums strictly positive. 

■ There exist already in the literature models of probabilistic processes, for 
I example the automata of Rabin f6], which are however non-compositional. An- 

other model which does includes compositionality is discussed in [5 . Our model 
is more expressive - the example we discuss in this paper cannot be described 
by the model in [S]. It is also, in our view more natural, and mathematically 
more elegant. We will make some comparison with the cited models in the last 
section of the paper. 

The idea of ^ was to introduce two-sided automata, in order to permit op- 
erations analogous to the parallel, series and feedback of classical circuits. For 
technical reasons which will become clear we first introduce weighted automata 
(where the weighting of a transition is a non-negative real number) and then 
Markov automata. We then show how to compose such automata, calculat- 
ing the probabilities in composed systems. An important aspect is the use of 
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conditional probability since, for example, composing introduces restrictions on 
possible transitions and hence changes probabilities. 

As an illustration of the algebra we show how to specify a system of n 
dining philosophers (a system with 12" states) and to calculate the probability 
of reaching deadlock in k steps, and we show that this probability tends to 1 as 
k tends to oo, using the methods of Perron- Frobenius theory. 

It is clear that the algebra extends to semirings other than the real numbers, 
and in a later work we intend to discuss examples such as quantum automata. 
An earlier version of this paper was presented at [S]. 

We are grateful for helpful comments by Pawel Sobocihski and Ruggero 
Lanotte. 

2 Markov automata 

Notice that in order to conserve symbols in the following definitions we shall use 
the same symbol for the automaton, its state space and its family of matrices 
of transitions, distinguishing the separate parts only by the font. 

Definition 2.1 Consider two finite alphabets A andB, containing, respectively, 
the symbols sa and Sb- A weighted automaton Q with left interface A and right 
interface B consists of a finite set Q of states, and an A x B indexed family 
Q = {Qa b) (aeA beB) of Q X Q matrices with non-negative real coefficients. We 
denote the elements of the matrix Q a.b by [Qa,b]q.q' (QtQ' £ Q)- We require fur- 
ther that the row sums of the matrix QeA,£B (and hence of Q ^ X^aeA beB ^a.b) 
are strictly positive. 

We call the matrix 

aeA.beB 

the total matrix of the automaton Q. 

Definition 2.2 Consider two finite alphabets A andB, containing, respectively, 
the .symbols £a and es . A Markov automaton Q with left interface A and right 
interface B , is a weighted automaton satisfying the extra condition that the row 
.sums of the total matrix Q are all 1 . That is, for all q 

q' aeA,beB 

We call [Qa.b]q.q' the probability of the transition from q to q' with left signal a 
and right signal b. 

The idea is that in a given state various transitions to other states are possi- 
ble and occur with various probabilities, the sum of these probabilities being 1. 
The transitions that occur have effects, which we may think of a signals, on the 
two interfaces of the automaton, which signals are represented by letters in the 
alphabets. It is fundamental not to think of the letters in A and B as inputs 
or outputs, but rather signals induced by transitions of the automaton on the 
interfaces. For examples see section 2.3. 

When both A and B are one element sets a Markov automaton is a Markov 
matrix. 
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Definition 2.3 Consider a Markov automaton Q with interfaces A and B. 

A behaviour of length k of CI consists of a two words of length k, one u = 
aia2 ■ • ■ Ok in A* and the other v = 6162 ■ • - bk in B* and a sequence of non- 
negative row vectors 



:ro,.-ri = xoQai,6i , •a;2 = a:iQa 



2,62 ) ' ' ' 7 '^k — '^k—l^a^ 



Notice that, in general, Xi is not a distribution of states; for example, in our 
examples often Xi =0. 

There is a straightforward way of converting a weighted automaton into a 
Markov automaton which we call normalization. 



2.1 Normalization 

Definition 2.4 The normalization of a weighted automaton Q, denoted N(Q) 
is the Markov automaton with the same interfaces and states, but with 



N(Q)a,, 



[Qaj 



[Qa.l 



q-q' J2q'eQ^^^q,q' ^q' eQ^aeA,beB \-^<l,b]g,j, 

To see that N(Q) is Markov, notice that the qth row sum of M{Q) is 



q' a.b _ 



J2q' J2a,b [Qa>b]g,g' 



1. 



Lemma 2.5 (i) If Q is a Markov automaton then N(Q) = Q. 

(ii) If Ca,b,q o-fc positive real numbers and Q and R are weighted automata 
(with the same interfaces A and B, and the same state spaces Q = R) such that 



[Qa,b]q,q' = Cq^a,b]q,q' 



then N(Q) =N(R). 
Proof, (ii) follows since 



E E [Q''."],,,' = E E Mq,q' = ^« E E [R''.^],,,' 



q' a,b 



q' a,b 



q' a,b 



and hence 



[Qa,b]g,g' 



[^a,b]g,q 



[Ro,b]g,g' 



□ 



An important operation on weighted automata is the power construction. 



2.2 The power construction 

Definition 2.6 If Q is a weighted automaton and k is a natural number, then 
form a weighted automaton Q'^ as follows: the states o/Q'^ are those o/Q; the 
left and right interfaces are A*' and respectively; e^k = {ea, ■ ■ ■ ,£a),£b'' = 
(sb, - ■■ ,£b)- ^/m = (ai, 02, • • • ,ak) & and t; = (5i, 62, • • • , 6fc) G B'' then 

(Q '^u,v Qai,6iQa2,??2 ' ' ' ^(^k,i>k' 

If Q is weighted and u — (ai, 02, • • • , Ofc) G A^ , v — (5i, 62, • • • ,bk) G B'', 
then [{Q'^)u,v]q,q' is the sum over aU paths from q to q' with left signal sequence 
u and right signal sequence v of the weights of paths, where the weight of a path 
is the product of the weights of the steps. 

Lemma 2.7 // Q is a weighted automaton then the total matrix of is the 
matrix power Q*^. Hence if Q is Markov then so is Q''. 

Proof. The q, q' entry of the total matrix of Q'^is 

E [i^')^Aq,q: 

ueA'' ,veB'' 

— [Qai,6i Qa2,ft2 ■ ' ' Qafc,fcfc]g,g' 

= E E ^'^"■l-'bl]q^q^[Qa2M]qi^q2' "l^ak,bk]q^__^y 

ueA'',veB'' 9l,---9fc-l 

= E EE [Q<»2,&2]gi,g2 ' " ['^°fc''"=]9fc-l,g' 

9l,---9fc-l ai,--- ,afc 6i,--- ,bk 

= E E ^^'^l'bl'\q,qi E [Q'^2'''2]gi,g2 " ' E '^k m] _ ,q, 
9lr--<?fc-l ai,6i a2,f>2 Qfc.tifc 

= [QQ---Q]q,q,- 

□ 

Definition 2.8 If Cl is a Markov automaton then we call Q*^ the automaton 
of k step paths in Q. FKe define the probability in Q of passing from state q to 
q' in exactly k steps with left signal u and right signal v to he [{Q^) ^ v^i^i' ■ 

It is important to understand the precise meaning of this definition. The 
probability of passing from state q to q' in precisely k steps, so defined, is the 
weighted proportion of all paths of length k beginning at q and ending at q' 
amongst all paths of precisely length n beginning at q. 

2.3 Graphical representation 

Although the definitions above are mathematically straightforward, in practice 
a graphical notation is more intuitive. We may compress the description of an 
automaton with interfaces A and _B, which requires Ax B matrices, into a single 
labelled graph, like the ones introduced in [4J. Further, expressions of automata 
in this algebra may be drawn as "circuit diagrams" also as in [3]. We indicate 
both of these matters by describing some examples. 
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2.3.1 A philosopher 



Consider the alphabet A = {t, r,e}. A philosopher is an automaton Phil with 
left interface A and right interfaces A, state space {1,2,3,4}, and transition 
matrices 



Phile, e = 



Phil 



t.e 







^ 
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The other four transition matrices are zero matrices. 
Notice that the total matrix of Phil is 



1 1 

2 2 





1 

! 

2 









1 

! 



2 J 



which is clearly stochastic, so Phil is a Markov automaton. 

The intention behind these matrices is as follows: in all states the philosopher 
does a transition labelled e,e {idle transition) with probability ^; in state 1 he 
does a transition to state 2 with probability | labelled t,e {take the left fork); 
in state 2 he does a transition to state 3 with probability ^ labelled e, t {take the 
right fork) ; in state 3 he does a transition to state 4 with probability | labelled 
r,s {release the left fork); and in state 4 he does a transition to state 1 with 
probability ^ labelled e,r {release the left fork). All this information may be 
put in the following diagram. 



t, r,s 
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2.3.2 A fork 



Consider again the alphabet A = {t,r,s}. A fork is an automaton Fork 
with left interface A and right interface A, state space {1,2,3}, and transition 
matrices 
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, Forke,t = 
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The other four transition matrices are zero. 

Fork is a Markov automaton since its total matrix is 

r 1 1 11 



The intention behind these matrices is as follows: in all states the fork does a 
transition labelled e,e {idle transition) with positive probability (either ^ or i); 
in state 1 it does a transition to state 2 with probability i labelled t, e {taken to 
the left); in state 1 he does a transition to state 3 with probability ^ labelled e, t 
{taken to the right); in state 2 he does a transition to state 1 with probability ^ 
labelled r,e {released to the left); in state 3 he does a transition to state 1 with 
probability ^ labelled e,r {released to the right). 

All this information may be put in the following diagram: 













1 X 






' ' 3 / 










e, r, 2 












c, 2 



2.4 Reachability 

For many applications we are interested only in states reachable from a given 
initial state by a path of positive probability. Given a Markov automaton Q and 
an initial state go there is a subautomaton Reach(Q,go) whose states are the 
reachable states, and whose transitions are those of Q restricted to the reachable 
states. 
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3 The algebra of Markov automata: operations 



Now we define operations on weighted automata analogous (in a precise sense) 
to those defined in [i]. 

Definition 3.1 Given weighted automata Q with left and right interfaces A 
and B, and S with interfaces C and D the parallel composite Q x R is the 
weighted automaton which has set of states Q x R, left interfaces A x C, right 
interface B x D, saxC — {'^A,£c), £bxd = {£b,£d) and transition matrices 



(Q X S)(„ 



(a,c},{b,d) 



Qa.b (8) Sc,d- 



This just says that the weight of a transition from {q, r) to {q' , r') with left 
signal (a, c) and right signal (6, d) is the product of the weights of the transition 
q q' with signals a and b, and the weight of the transition r ^ r' with signals 
c and d. 

Lemma 3.2 //Q and R are weighted automata then 

N(Q X R) =N(Q) X N(R). 
Hence if Q and R are Markov automata then so is Q x R. 
Proof. 



N(Q X R), 



{a,c),{b,d) 



[Qa,b]q,q> {^c,d] 



{q,r),iq' ,r') 



[Qaj 



'1,1 



X]q',(a,6) \^0-Aq^qi Z]r',(c,d) P 



N(Q)(a 



(a.b) 



c,d\ 



(l,r),{q' ,r') 



l(c,d) 

= _(N(Q) X N(R))(,,,)^(,,,) 

For the second part notice that if Q and R are Markov then 

Q X R = N(Q) X N(R) = N(Q x R) 

which implies that Q x R is Markov. 
□ 



Definition 3.3 Given weighted automata Q with left and right interfaces A 
and B, and R with interfaces B and C the series (communicating parallel) 
composite of weighted automata QoR has set of states Q x R, left interfaces 
A, right interface C , and transition matrices 



{Qo^)a.c = ^Qa.b®^b 



beB 



Lemma 3.4 o R) o S = Q o (R o S). 
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Proof. This follows from the fact that (g) is associative.. 

□ 

It is easy to see that QoR is not necessarily Markov even when both Q and 
R are. The reason is that the communication in the series composite reduces 
the number of possible transitions, so that wc must normalize to get (condi- 
tional) probabilities. However in a multiple composition it is only necessary to 
normalize at the end, because of the following lemma. 

Lemma 3.5 N(N(Q) o N(R)) = N(Q o R). 

Proof. 



-I (q,r),(g',r') '-^ '''^ 



fees 



Sg' So, 6 [Q«.b]q,g' Sr' Sb,c [f^b.Jr.r'' 



= Co 



(Q°R)a,. 



beB 

, where 

(,q,r),{q',r') 

depends only on q, r. 



Hence by the lemma 2.5 above 

N(NQ o NR) = N(Q o R). 

□ 

Definition 3.6 // Q and R are Markov automata, Q with left interface A and 
right interface B, R with left interface B and right interface C then the series 
composite of Markov automata Q • R is defined to 6e Q • R = N(Q o R). 

Theorem 5.7 fQ • R) • S = Q • (R • S). 

Proof. 

(Q-R)-S = N(N(QoR)oS) 

= N(N(Q o R) o N(S)) since S is Markov 

= N((Q o R) o S) = N(Q o (R o S)) by 3.6 and 3.6 

= N(N(Q) o N(R o S)) by 3.6 

= N(Q o N(R o S)) = Q • (R • S) since Q is Markov. 

□ 

Theorem 3.8 If Q and R are Markov automata then 

(Q X R)*= = Q*^' X R'^ 
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Proof. 



(Q X R)") 

(Qai,bi«)Qo2,b2® ■ ■ ■ Qafc,bJ(Rai,b;®R4,b^ ■ ■ ■ ®Ra;^,f,'J 

□ 

Remark. It is not the case that if Q, R are Markov then (Q ■ R)'^ = Q'^ • R*^. 
The reason is that normaUzing length k steps in a weighted automaton is not 
the same as considering k step paths in the normahzation of the automaton. 
Next we define some constants each of which is a Markov automaton. 

Definition 3.9 Given a relation p C A x B such that (ea, £b) £ p we define a 
Markov automaton p as follows: it has one state * say. The transition matrices 
[pa,b] are 1 x I matrices, that is, real numbers. Let \p\ be the number of elements 
in p. Then pafi = */ p relates a and b, and pa^t = otherwise. 

Some special cases, all described in have particular importance: 

(i) the automaton corresponding to the identity function 1a, considered as a 
relation on A x is called 1a', 

(ii) the automaton corresponding to the diagonal function A : A A x A 
(considered as a relation) is called A^; the automaton corresponding to 
the opposite relation of A is called Va- 

(iii) the automaton corresponding to the fmiction twist : A x B ^ B x A is 
called twist^^s- 

(iv) the automaton corresponding to the relation rj = {{*, (a, a); a G A} C 
{*} X (A X A) is called t]a', the automaton corresponding to the opposite 
of rj is called eA- 

3.1 The dining philosophers system 

Now the model of the dining philosophers problem we consider is an expression 
in the algebra, involving also the automata Phil and Fork. The the system of 
n dining philosophers is 

DF„ = r]Ai{Phil Fork • Phil Fork Phil • Fork) x 1^) • e^, 

where in this expression there are n philosophers and n forks. 

As explained in 4J , we may represent this system by the following diagram, 
where we abbreviate Phil to P and Fork to F. 
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Let us examine the case when n ~ 2 with initial state (1,1,1,1). Let Q 
be the reachable part of DF2. The states reachable from the initial state are 
(71 = (1, 1, 1, 1), 92 - (1, 3, 3, 2), 93 = (3, 2, 1, 3), 94 = (1, 1, 4, 2), 95 = (4, 2, 1, 1), 
(76 = (1, 3, 2, 1), (77 = (2, 1, 1, 3), (78 = (2, 3, 2, 3) (gs is the unique deadlock state). 
The single matrix of the automaton Q, using this ordering of the states, is 



\ 









I 

2 





1 

3 









I 

3 















i 



i i i 

"444 



1 




1 

? 

3 

1 



Calculating powers of this matrix we see that the probability of reaching 
deadlock from the initial state in 2 steps is ||, in 3 steps is and in 4 steps 



4415 
6912 • 



4 The probability of deadlock 

The idea of this section is to apply Perron-Frobenius theory (see, for example) 
[7] to the Dining Philosopher automaton. However, for convenience, we give the 
details of the proof of the case we need, without refering to the general theorem. 

Definition 4.1 Consider a Markov automaton Q with input and output sets 
being one element sets {e}. A state q is called a deadlock if the only transition 
out of q with positive probability is a transition from q to q (the probability of 
the transition must necessarily be 1). 

Theorem 4.2 Consider a Markov automaton Q with interfaces being one ele- 
ment sets, with an initial state qo- Suppose that 

(i) Q has precisely one reachable deadlock state, 

(ii) for each reachable state, not a deadlock, there is a path with non-zero 
probability to qo, and 

(Hi) for each reachable state q there is a transition with non-zero probability 
to itself. 

Then the probability of reaching a deadlock from the initial state in k steps tends 
to 1 as k tends to infinity. 

Proof. Let R = Reach(Q, (70). Suppose R has m states. Then in writing the 
matrix R we choose to put the deadlock last, so that R has the form 



R 



S T 

1 



where S is (m — 1) x (m — 1) and T is (m — 1) x 1. Now 







1 
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for some matrix Tfc. Condition (i) implies that there is a path with positive 

probability (a positive path) from any non-deadlock state to any other in R. 
Condition (ii) implies that if there is a positive path of length I between two 
states then there is also a positive path of all lengths greater than I. These two 
facts imply that there is a fco such that from any non-deadlock state to any other 
state there is a positive path of length fco- For this ko the matrix T^gis strictly 
positive. This means that the row sums of 5*^° are strictly less than 1. But the 
eigenvalues of a matrix are dominated in absolute value by the maximum of the 
absolute row sums (the sums of absolute values of the row elements) . Hence the 
eigenvalues of S'^° and hence of S all have absolute value less than 1. But by 
considering the Jordan canonical form of a matrix whose eigenvalues all have 
absolute values less than 1 it is easy to see that S*^ tends to as fc tends to 
infinity. Hence Tfc tends to the column vector all of whose entries are 1. Hence 
the probability of reaching the deadlock from any of the other states in k steps 
tends to 1 as fc tends to infinity. 
□ 

Corollary 4.3 In the dining philosopher problem DF„ with qo being the state 
(1, 1, ■ • • , 1) the probability of reaching a deadlock from the initial state in k 
steps tends to 1 as k tends to infinity. 

Proof. We just need to verify the conditions of the theorem for the dining 
philosopher problem. It is straightforward to check that the state 

(2,3,2,3,--- ,2,3) 

in which the philosophers are all in state 2 and the forks in state 3 is a reachable 
deadlock.lt is clear that in any state q there is a positive transition to q, since 
each component has silent moves in each state. We need only check that for 
any reachable state other than this deadlock that there is a positive path to the 
initial state. Consider the states /i, /2 of two forks adjacent modulo n, and 
the state p of the philosopher between these two forks. Examining the positive 
paths possible in two adjacent forks and the corresponding philosopher we see 
that the reachable configurations are limited to (a) /i = 1, p = 1, /2 = 1, (b) 
/i = 1, p = 1, /2 = 3, (c) /i = 1, p = 4, /2 = 2, (d) /i = 2, p = 1, '/2 = 1, (o) 
/i = 2, p = 1, /2 = 3, (f) /i = 3, p = 2, /2 = 1, (g) /i = 3, p = 2, /2 = 3, (h) 
fi — 3, p — 3, f2 = 2, (i) /i = 2, p = 4, /2 = 2. We will show that in states 
other than the deadlock or the inital state there is a transition of the system 
which increases the number of forks in state 1. Notice that in a reachable state 
the states of adjacent forks determine the state of the philosopher between. 
Consider the possible configurations of fork states. We need not consider cases 
all forks are in state 1 (initial), or all in state 3 (the known deadlock). Given 
two adjacent forks in states 3, 2 there arc transitions which only involve this 
philosopher and the two forks (apart from null signals) which rcsidt in one of 
the forks returning to state 1 (the philosopher puts down a fork that he holds). 
This is also the case when two adjacent forks are in states 1,2 or 2,2 or 3, 1. 
But in a circular arrangement other than all I's or all 3's one of the pairs 1,2 
or 2, 2 or 3, 1 or 3, 2 must occur. 
□ 

Remeirk. Notice that in the proof of the corollary we did not use the specific 
positive probabilities of the actions of the philosophers and forks. Hence the 
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result is true with any positive probabilities replacing the specific ones we gave 
in the description of the philosopher and fork. In fact, different philosophers 
and forks may have different probabilities without affecting the conclusion of 
the corollary. 

5 Concluding remarks 

5.1 The algebra of automata: equations 

There is much more to say about the algebraic structure and its relations with 
other fields. We have mentioned above some equations which are satisfied, and 
here we mention one more. 

Lemma 5.1 The constants A^, \I a satisfy the Frobenius equations fTI, namely 
that 

{Aa X 1a) ■ {1a X Va) = • A^. 
The proof is straightforward. 

5.2 Comparisons 

According to Rabin a probabilistic automaton on an alphabet E consists of a 
set of states Q and a family of stochastic transition matrices [Pa]^,^' (a € S; 
q,q' Q). A distribution of states is a row vector with non negative real entries 
whose sum is 1. A behaviour corresponding to an initial state distribution xq, 
and an input word u = 0102 • • • Ofe , is a sequence of state distributions 

Xo,Xi = XQPa^,X2 = XoPa^,- ■■ ,Xk = Xk-lPa^- 

This is a non-compositional model, and it immediately clear that the meaning of 
the alphabet in Rabin is quite different from the meaning of the alphabets for our 
Markov automata. For Rabin the letters are inputs which drive the automaton 
Q - for a given state q and a given input a the sum of the probabilities of 
transitions out of g is 1. We are able to describe the same phenomenon by 
considering a second automaton R whose signal on the interface drive Q, which 
of course introduces conditional probabilities. From our point of view Rabin's 
probabilities are conditional ones resulting from the knowledge that an input 
a occurs. Another difference is that every transition in every state in Rabin's 
automata produces a distribution of states. However it is crucial in our model 
that actions are not necessarily defined in all states; or if they are defined they 
may be only partially defined. For example, the fork in state 2 has no transitions 
labelled t, e; it cannot be taken again when it is already taken. The fork in state 
1 may be taken to the left with probability ^. 

The second model we mentioned [5] considers a generalization of Rabin's 
model (and hence different from ours) which is influenced by concurrency theory. 
It has a form of composition, and as usual with models related to process alge- 
bras the composition involves an underlying broadcast (and hence interleaved) 
communication, and does not involve conditional probability. As a result it 
is not possible to describe our example, in which, in a single step all philoso- 
phers may take their left fork. Instead, it is straightforward to model broadcast, 
interleaved models using our algebra [5], using in particular the component A. 
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Further, 5 has a much more hmited algebra than that presented here; for 
example, multiply simultaneous signals (together with the synchronization on 
some of the signals) are not available. 

In our view interleaving destroys the realism of the model. For example, 
to reach deadlock in the dining philosopher problem requires a sequence of 
actions, as philosophers take the forks one by one. This results in quite different 
probabilities. 
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